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ABSTRACT 


Several NASA regulations specify that formal risk analysis be 
performed on a program at each of several milestones as it moves toward 
full-scale development. Program risk analysis is discussed as a systems 
analysis approach to risk, an iterative process (identification, assess- 
ment, management), and a collection of techniques. These techniques, 
which range from extremely simple to complex, network-based simulation, 
were surveyed. A Program Risk Analysis Handbook was prepared in order 
to provide both analyst and manager with a guide for selection of the 
most appropriate technique. 

Various researchers have verified that 85-90% of the risk in 
complex, technological systems development originates in the technical 
definition of the system. Risk may be assessed on cost, schedule, or 
performance individually; the preferred approach is to treat these as 
dependent random variables and perform an integrated risk assessment. 

All program risk assessment techniques were shown to be based on elici- 
tation and encoding of subjective probability estimates from the various 
area experts on a program. Techniques to encode the five most common 
distribution-types were given. Then, a total of twelve distinct ap- 
proaches to risk assessment were given. For each approach we identified 
the steps involved, good and bad points, time involved, and degree of 
computer support needed. 

We discuss why risk analysis should be used by all NASA program 
managers. How to establish a risk analysis capability and some of the 
special difficulties in performing a risk analysis were related. Tools 
available at NASA/MSFC were identified, along with commercially 
available software. Both an extensive bibliography (150 entries) and a 
program risk analysis check-list were provided. 

Recommendations are to: 

1. Perform integrated cost/risk assessment on each program, prior to 
each RFP release. 

2. Require contractors to perform quantitative risk assessments during 
proposal preparation and after contract award. 

3. Select (or hire) a full-time risk analyst in Program Planning, with 
responsibilities for applications, methods development, interface 
with other centers, and consulting to program and engineering 
managers . 
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INTRODUCTION 


The article n GRO Project Beset by Complications" appeared in the 
6/14/87 Huntsville Times. The NASA/GSFC Project Manager, Jeremiah 
Madden, stated "the sheer magnitude and complexity of the GRO program 
overwhelmed managers and engineers and obscured some of the program T s 
finer details." These complications occur on all space and defense 
projects, especially those that use unproven technology, attempt a new 
mission, and/or scale up (or down) the size of the craft used — for 
example, the C5A, the Trident Submarine, Space Telescope, and the 
Stealth Bomber. The specific problems encountered on the GRO project 
are typical: 

o manufacturing processes not well-understood 

o manufacturing problems due to materials faults/availability 
o lack of trained manufacturing work force 

o unexpected electromechanical interference between instruments, 
once integrated 

o redesign of components and tooling 

These problems led to a modest cost growth of $380 million to $500 
million, and a schedule slip from May f 88 to early 1990. Space Tele- 
scope cost growth is $500 million to $1.4 billion. The average cost 
growth observed in both NASA and DoD projects, from the start of Full- 
Scale Development (FSD) to the completion of prototype production, has 
been in the range 30-40%. 

This type of track-record for Federal acquisition of large-scale 
systems has led to Congressional skepticism, public outrage, and occa- 
sionally loss of support for the continuation of the project. Since the 
’60s, DoD and NASA project managers have sought management techniques 
that will help them control cost growth and schedule slippages. Major 
General John R. Guthrie, USA, stated at the 1970 DoD Project Managers 
Conference that: 

"the most rudimentary sort of good risk analysis might have enabled 
us to avoid most of the pitfalls we have encountered. By rudimen- 
tary I mean — did we identify those items which were new and 
identify the impact on overall system performance if that partic- 
ular component or subsystem were to experience difficulty?" 

Program risk analysis is an iterative process (Figure 1) for 
identifying, quantifying, and managing the uncertainties associated with 
complex design and development programs typical to NASA. Others have 
defined risk analysis as a systems analysis approach to risk, or as a 
collection of techniques to identify, quantify, and manage risk. In 
contrast to techniques for quantifying operational risk (e.g., failure 
modes and effects analysis, fault tree analysis, reliability analysis), 
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Figure 1 Risk Analysis Iterative Process 



program risk analysis deals with program cost, schedule, and technical 
performance estimates. The key question to be answered by risk analysis 
is what are the distributions of probability on the mature (achieved) 
values of each of these three random variables. In some risk analyses, 
two of these variables (say, performance and schedule) are thought of as 
fixed. Then the probability distribution on the mature value of the 
other (cost) is derived. However, in a comprehensive risk analysis, the 
dependence among these variables is assessed and uncertainty in one 
affects the assessment of risk in the other two. 

NASA NMI 7100. 14A (Major System Acquisition) calls out risk eval- 
uation as a criteria second only to performance for initiating FSD . MMT 
7110.1 requires risk analysis and cost risk assessment for both formal 
project pre-development reviews. Program risk analysis is useful to 
program managers as both a source of information on the program and as a 
decision-aiding tool. The reason is that it is a formal, systematic, 
and documented approach to dealing with uncertainty, versus M seat-of-the 
pants" dealing with problems as they arise. It can be used in both 
Phases A and B, as requirements and design configurations evolve, for 
the purpose of early identification and resolution of technical uncer- 
tainties. Classic risk resolution strategies are parallel development, 
design/operations trade-offs, and development of back-up solutions. It 
is probably of most use for assessing the potential for cost and sched- 
ule slippages in programs moving into Phase C/D, because this is the 
point at which significant resource commitments will be made. Risk 
analysis can be applied to subsystems or instruments, individual tech- 
nologies, manufacturing processes, and other elements that make up a 
program; however, its true value is in synthesizing these multiple 
uncertainties . 



OBJECTIVES 


Each task team leader and project manager (PM) at NASA/MSFC should 
become aware of the valuable information available from a program risk 
analysis. The risk analysis process is iterative and risk analysis 
should not be viewed as a one-time, check-the-box type activity. This 
handbook is written with the view of the PM as the consumer for risk 
analysis, and a range of options is provided so that the appropriate 
type of analysis, at the right level of detail, may be requested. 

This handbook is prepared as a guide and reference source for any 
NASA employee who is requested to perform a risk analysis in support of 
the PM. This individual will be referred to as "the risk analyst," 
although he /she may be a cost analyst, schedule analyst, program ana- 
lyst, engineer, or scientist. An entire range of risk analysis tools 
will be provided, along with some guidance for selecting the appropriate 
technique for a given situation. However, it is always necessary that 
the risk analyst apply his judgment when initiating a risk analysis at 
the request of a PM. For example, he must decide what technique is 
appropriate, given the time available and the software tools he has at 
his disposal. Another key question is how much access to and coop- 
eration from program personnel the analyst can expect; no meaningful 
risk analysis can be generating without repeated, probing discussions 
with practically all program personnel in order to identify technical 
uncertainties and their potential cost and schedule impacts. Good 
relations between risk analyst and technical team members is essential 
if a valid, useful risk analysis is to be conducted. The only alterna- 
tive, and one that works well, is for the risk analyst to be part of an 
ad-hoc team of experts, independent of the project, whose job it is to 
review the project and report to higher authority on its findings. 

The selection and support of a risk analyst (or risk analysis 
group) is an important step for large industry/government design orga- 
nizations and laboratories. The risk analyst is the alter-ego of top 
management in his evaluation of a project, much as the quality assurance 
analyst is for a production facility. He is often feared, avoided, 
circumvented, and detoured. Other engineers generally view the risk 
analyst as a nuisance who will take their valuable time, produce nothing 
new, and perhaps misrepresent their professional judgment. Although it 
is desirable for the risk analyst to be involved early in programs, and 
have an established relationship with both PM and project personnel, the 
fact is that risk analysis is usually a last-minute effort, performed on 
an ad-hoc basis, prior to some major decision/presentation. The risk 
analyst is often not at all familiar with the technology or program 
being analyzed. He must therefore: 

1. Educate himself quickly. 
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2 . 


Acquire the data. 


3. 


Use the data in some pre-developed model. 


4. Present recommendations based on model output. 

The above discussion clearly reveals that conducting a risk analv 
sis 1S not an easy job. The risk analyst must be an indfvidual of ^ 

relations^and^ “ ed ? C f ion ’ technical/program experience, human 

fii a k reco S n ition of management needs. Many of these positions 
filled by individuals with graduate-level training in statistics 
operations research, or systems analysis. An undergrfduate degree i^ 
engineering or hard sciences helps, but is not necessary Individuals 
a ° en *j hr alled wi th math models and/or computer techniques generally 
aren t good risk analysts. The interaction viih project personnel tie 
collection and verification of data, and preparation of fading i^to a 
format useful for management are the key activities of the risk analyst. 
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TERMINOLOGY 


Several general references on program risk analysis are given in 
the Bibliography. The definitions presented below are stated in broad 
terms and would be accepted by anyone working in the program risk 
analysis area. Of course, certain organizations and programs within 
organizations adopt more specific definitions for terms such as risk 
area " — in fact one of the first jobs for a risk analyst newly assigned 
to a program is to work with the PM on an agreeable set of definitions 
and working groundrules. Note also that more specialized terminology is 
used in health and environmental risk analysis, and these terms are not 
appropriate for program risk analysis. 

Definitions 

Risk -- the probability of undesirable future consequences of actions 
(inaction) taken today. Thus risk has a temporal element, and is a 
function of both probability and consequence. 

Program risk — the probability that the actual cost, time, or perfor- 
mance of a system will fail to match predictions. Also, the degree by 
which such predictions are missed and the associated consequences. 

Potential problem — an identified, but not yet occurring problem that 
if actualized, will impose unplanned resource demands, rescheduling, 
and/or degraded performance, quality, or safety margins. 

Risk area — a collection of related potential problems. Also a common 
source of several potential problems. 

Potential problem analysis (risk identification) identification of 
risk areas and the sequence of interrelated potential problems that stem 
from them. Also can include identification of immediate cost, schedule, 
and performance impacts of potential problems, recognizing that poten- 
tial problems may actualize at one of several levels of severity and 
that there is a probability associated with each level. 

Risk assessment — using the information from risk identification, and 
one or more quantitative techniques to synthesize the information, to 
create an overall assessment of program cost, schedule, or technical 
risk and also an assessment of the risk contributed by each risk area. . 
May include ranking risk areas by severity or timing in order to identi- 
fy a course of action. 

Risk management — identifying alternatives, selecting an approach, and 
taking action in order to reduce risk to levels deemed acceptable by the 
organization. Action may be directed at risk reduction or in trading 
one type of risk for another. In some instances, work around plans 
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and/or contingency budgets are defined as back-up solutions to the 
selected risk reduction approach. 

Probability — the relative frequency of an outcome of a repeatable, 
observable experiment. Also, a measure between 0 and 1 assigned to each 
outcome of an experiment based on its relative frequency. 

Subjective probability — a measure of the lack of information an 
organization or an individual has about the actual outcome of some 
future experiment. Essentially, it is a "degree of belief" measure 
based on human experience and reasoning, as opposed to a "frequency of 
occurrence" measure. 

Probability encoding — a process whereby the lack of information of an 
expert is quantified as a subjective probability distribution on a state 
variable, developed under specific assumptions, in a scientifically 
correct way, with as much accuracy as is justifiable. Accuracy can be 
increased by spending more time per encoded variable, or by combining 
the opinions of several experts. 
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RISK ANALYSIS METHODOLOGY 


A 95-page Progr am Risk Analysis Handbook was prepared as the 
primary product of this fellowship. This handbook will be published as 
a NASA/MSFC Technical Memorandum (TM) . The contents will be summarized 

here . 

All risk assessment techniques use subjective probability distri- 
butions as input. The process of interviewing a technical expert an 
determining the nature of the uncertainty in a variable of interest s 
referred to as "probability encoding." We discuss this process and give 
algorithms to encode the density function of five distribution types: 
uniform, triangular, beta, Weibull, and normal. 

Twelve distinct techniques for risk assessment were identified and 
discussed. We grouped these techniques into three classes: 

o Quick Risk Assessment Techniques , which require 1-2 days to 
implement and use point probability estimates (risk factors) 
as inputs. These methods require only a hand-held calculator, 

and include: 


o 


- Equi-risk contours method 

- Risk factor method (RFM) 

- Probabilistic event analysis (PEA) 

- Probabilistic Evaluation and Review Technique (PERT) 

- Analytic Cost Risk Method 

- Method of Moments 

Standard R isk Assessment Techniques , which require 1-2 weeks 
to implement and are based on Monte Carlo simulation. These 
techniques require at least a microcomputer, and include: 


o 


- Simulation of the Critical path 

- Simulation of the Project Network 

- Simulation of the Cost Model 

- Simulation of a Performance Model 


Integrated Risk Assessment Techniques, which require 1-2 
months to Implement and are performed using a network-based, 
simulation package such as GERT, VERT, or RISNET. The tech- 
niques are: 


- Integrated Cost/Schedule Risk Assessment, based on direct 
evaluation of cost and time uncertainty on each activity. 
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- Integrated Technical/Cost/Schedule Risk Assessment, based on 
simulation of the occurrence of technical problems and their 
time/cost penalties. 

Note that only in the integrated techniques are time and cost 
treated dependently. A full discussion of each of these techniques and 
related reference material may be found in the Program Risk Analysis 
Handbook , The techniques currently used at NASA/MSFC are indicated by 
an asterisk in Figure 2 below. 



USi 6# redHMduli 1 

TIME TO 
IMPLEMENT 

SUBJECTIVE FACTORS AND/OR 
POINT PROBABILITIES 

SUBJECTIVE PROBABILITY A 
ANALYTICAL METHOOS 

SUBJECTIVE PROBABKJTV A 
MONTE CARLO SIMULATION 

1 * 2 DAYS 

1 • 2 WEEKS 
1 - 2 MONTHS 

t 

• EOUI-RISK CONTOURS 

• RISK FACTOR METHOO 

• PROBABILISTIC EVENT 
ANALYSIS 

1 

f 

• PERT 

• ANALYTIC COST 
RISK 

B METHOO OF MOMENTS 

l 

CRITICAL PATH SIMULATION* 

SIMULATION OP COST MOOEL* 
SIMULATION OP SCHEDULE 
NETWORK 

SIMULATION OP PERFORMANCE 
MOOEL 

INTEGRATED COST/SCHEDULE 
NETWORK SIMULATION 

INTEGRATED TECHNICAUCOSTl 
SCHED. NETWORK SIMULATION 


* - IN USE AT NASA/MSFC 


INOUSTRY/ 

-GOVKRNMT. 

STAMOAAO 


NETWORK* 
BASED 
ST ATI* OF 
•THE -ART 


Figure 2 Summary of Risk Assessment Method- 
ology Alternatives 
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CONCLUSIONS AND RECOMMENDATIONS 


Based on a thorough literature search, the range of quantitative 
methods for program risk assessment have been presented. As shown in 
Figure 2 twelve distinct alternatives were identified. These are 
discussed in detail in the Program Risk Analysis Handbook , prepared as 
the major product of this fellowship. All methods are based on the 
Bayesian view of probability; they differ in how subjective probability 
is collected (level of detail, assumptions, distribution types, etc.) 
and how these probabilities are combined into an overall assessment of 
uncertainty. Although "a risk assessment" can be done in a matter of 
several days, the truly comprehensive risk methods treat technical, 
cost, and schedule risks in an integrated (network-based) fashion and 
require at least one month of up-front development. The management 
b enefits of integrated, n etwork-based methods are worth the expense and 
waiting-time for the initial model output . 

The recommendations for NASA/MSFC based on discussions with Program 
Planning Office personnel and the contents of the handbook are: 


Commit to performing integrated cost/schedule risk assessment 
on each program prior to releasing RFPs for the phase in 
question. Quick risk assessments are, of course, appropriate 
in certain circumstances. 


2. Require contractors to perform quantitative risk assessments 
as part of their proposal preparation effort. Require that 
these assessments be submitted as part of the technical volume 
or as a separate volume, or back-up document. Be explicit 
that meaningless LOW-MEDIUM-HIGH risk ratings are not accep- 
table, and that integrated methods are preferred. Require 
risk analysis be part of the systems analysis /management 
process after contract award. 

3. Select (or hire) a full-time risk analyst to be stationed in 
Program Planning with the following responsibilities: 

o Perform risk analyses on all PD studies, with early involve- 
ment with PM and study team. 

o Write risk analysis requirements for all RFPs. 

o Develop and document databases, questionnaires, and methods. 

o Plan evolution of tools, either in— house development or 
outside acquisition. 

o Train project control personnel to perform risk analyses. 
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o Interface with other centers. 


o Consult with PMs , chief engineers on use of risk analysis on 
their project. 

Commit to investing the time and money to build a 
state-of-the-art capability in program risk analysis at 
NASA/MSFC . 

o Give selected analyst one year to build background, learn to 
use tools you have (ARTEMIS, SLAM, and SAM), and to review 
available methods and computer packages. 

o Consider purchase of network simulation package designed for 
risk analysis. 

- RISNET 

- MICRO-VERT 

Inform technical personnel in PD, and lab personnel supporting 
PD, about what risk analysis vis and how they may be involved. 
Perhaps include some training in basic statistical concepts 
(classical and Bayesian) and generally encourage team-work, 
cooperation in generation of risk information. 

As experience is gained, consider expanding this handbook to 
include : 

- risk identification methods 

- risk management methods 

- lessons learned 

- case histories 

Consider expanding from one risk analyst to a risk and 
decision analysis group. 
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